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ORTS purpose statement 


Ensure the success of current and 
future Research and Technology 

Development Efforts. (Even when “ success ” 

may be learning from a failure of the particular science or technology 

under study. ) 
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ORTS Charter 


The OSMA Research and Technology Strategy Team shall: 

• Look for new or streamlined approaches to provide risk information to Research and Development (R&D) 
researchers. NASA's Safety and Mission Assurance (SMA) community must provide input to R&D efforts in a 
manner which helps to build safety, quality, and reliability into new technology development without impeding 
progress. Consider any differences in approaches that may be needed for R&D in specific areas; e.g., 
aeronautics, ground operations, robotics, modeling and simulation, space flight. 

• Benchmark NASA successes in this area as well as other government agencies and possibly industry (e.g. ; 
Department of Defense, Federal Aviation Administration, auto industry, aviation industry, and Defense 
Advanced Research Projects Agency) and seek out strategies for incorporation of Safety, Reliability, and 
Quality Assurance (SR&QA) into the research and technology processes. This may include: 

types of SR&QA requirements levied on Principle Investigators and/or test rigs and facilities 

assessment and evaluation of risks associated with proposed tests and or demonstrations. 

level of SR&QA oversight provided (or not) 

participation in design reviews and acceptance reviews 

process for verification and validation of requirements 

process for testing 

process for acceptance of risk 
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ORTS Charter (Cont.) 


• Provide strategies for supporting upgrades and certifications for test 
cells, facilities, and test areas to support demonstrations and prototype 
tests. 

• Provide strategies for better supporting simulations and model 
development. 

• Provide SR&QA approaches needed for transitioning from prototypes to 
actual development activities with operational products and systems. 

• Identify possible opportunity areas for future SR&QA research including 
tools development and process improvement. 
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ORTS Team Process Overview 


• Bi-monthly to weekly Telecon/Web-Exs with discussions, minutes, and action items 

• Created and used PBMA Secure Site 

• Membership including SMA Center representatives w. R&T experience, HQ OSMA, 
OSMA Tech Fellows, and researchers 

• Reached out to Center researchers to gain better understanding of their perspective 

• Reviewed current policies 

• Benchmarked internal and external SMA needs and processes for R&TD 

• Discussed with OCT their expectations vs SMA’s of SMA's role 

• Worked with OCT to understand their needs and perspective 

• 2 face to face working meetings, at start and to finalize work and interviews 

• Some Center Technology Road Maps examined and discussed to help define SMA 
roles 

• Gathered LL and both success and failure Stories from past research efforts. 
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NASA Technology Readiness Levels 

• TRL 1 Basic Principles observed and Reported 

• TRL 2 Technology concept and/or application formulated 

• TRL 3 Analytical and experimental critical function and/or characteristic 
proof-of-concept 

• TRL 4 Component and/or breadboard validation in laboratory environment 

• TRL 5 Component and/or breadboard validation in relevant environment 

• TRL 8 Svstem/subsystem model or prototype demonstration in a 
operational environment (ground or space) 

• TRL 7 System prototype demonstration in an operational environment 

• TRL 8 Actual system completed and “Flight qualified” through test and 
demonstration (ground, airborne or space) 

• TRL 9 Actual system “flight proven" through successful mission operations 
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Facility Safety and 
usage 

requirements 


NPR 7120.8 
Implementation 


NPR 7120,5 
Implementation 


• General, Graduated SMA Coverage 


Technology Readiness Levels Summary: 

TRL 1 Basic Principles observed and 
Reported 

TRL 2 Technology concept and/or 
application formulated 

TRL 3 Analytical and experimental critical 
function and/or characteristic proof-of- 
concept 

TRL 4 Component and/or breadboard 
validation in laboratory environment 

TRL 5 Component and/or breadboard 
validation in relevant environment 


Center Institutional Safety 
Requirements 

'High Risk Test Facilities 

•Labs 

•OSHA 

•Industrial Hygiene 
•Fire/Life Safety 
•Emergency Preparedness 
•Awareness and insight both ways 


Project dependent, 
*Area targeted for SMA 
improvements 


TRL 6 System/subsystem model or 
prototype demonstration in a operational 
environment (ground or space) 

TRL 7 System prototype demonstration in 
an operational environment 

TRL 8 Actual system completed and "Flight 
qualified" through test and demonstration 
(ground, airborne or space) 

TRL 9 Actual system "flight proven" through 
successful mission operations 


NASA and Center SMA 
Requirements 

•System Safety 
•Quality Assurance 
•R&M 

•Software Assurance 
•Risk Management 





TRL Level Discussion 


• TRLs 1-3: mostly paper concepts/theories, simulations, possibly Facility or 
bench top 

- SMA informs PMs, Pis of risk concerns and process, thinking ahead to safety, 
quality, & reliability needed for moving to demonstration phase 

- SMA becomes informed about the work in progress so can prepare if moves 
forward 

• TRLs 4-6: use of Facilities and some experimental build up, proof of 
concepts, where decision to go to flight demonstration is made 

- SMA informs PMs, Engineers and Scientists of potentail risks, quality, safety and 
reliability expectations especially if use of our facilities but also general safety 
expectations 

- SMA participates on varied basis based on need/risk which has to be assessed 
individually 

- SMA gains insight and prepares for possible advancement of technology 

• TRL 6-9: Flight or Ground Demonstrations 

- Project and SMA meet 7120.8 and/or 7120.5 flight development requirements for 
engineering and SMA 
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Executive Summary: Findings 

Findings: 

1 . Agency Safety & Mission Assurance (SMA) needs to develop a consistent yet flexible 
approach to supporting Agency Research and Technology (R&T) efforts which 
incorporates Lessons Learned and Best Practices from Centers, Government, & 
Industry 

2. Area for greatest improvement in SMA involvement is OCT’s Game Changing 
Technologies /TRLs 4-6. 

a) Lower TRL’s, (theories, papers, models, etc.) SMA not needed or covered via strict safety rules 
for NASA on-site facilities 

b) Higher TRL’s, Fight Demonstrations on NASA payloads & aircraft must meet NPR 7120.5. 

3. SMA R&T approach not just based on TRLs, must consider level of risks to personnel, 
public, facilities, and major equipment, both current risks and potential future risks as 
the technology may move forward 

Consider added SMA support in certain technology domains (e.g. fuels, nuclear, electrical, 
pressure vessels, biology, etc.); complexity; flight vs non-flight 

4. Early involvement with R&T efforts and their sponsors (including OCT) is essential if only 
for awareness on both sides & to prepare to support later 

5. Education needs to go both ways: SMA to R&T and R&T to SMA 
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Executive Summary: Recommendations 


Recommendations: 

1 . Create targeted guidance and training: 

a) For SMA personnel: understanding and discernment of R&T culture and needs 

b) For researchers and PMs of R&T: SMA support options, SR&QA and risk requirements 
awareness 

2. Updates to NPR 7120.8, NPR 7120.5 and OSMA polices for better coverage of 
TRLs 4-6 

3. Updates to NPR 7120.8 to better incorporate/ involve SMA awareness and 
consideration in early TRLs (from conceptual phase to finish) 

4. Need multi-tiered points of contact established to better partner with OCT, 
Mission Directorates, and Centers with a focus on research and technology 

5. Establish SMA as a key player in technology developments that are identified as 
having a stated purpose of improving SRM or QA characteristics 

6. Fund OSMA from separate pot of money, i.e. CMO, for lower TRLs, so as not to 
impact research but still allow SR&QA the opportunity to gain awareness early on 
and provide training. 
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Executive Summary: Benchmarking Major Discovery 


• Biggest Surprise from Benchmarking: National Labs, 

AFRL, & even DARPA (to some extent) have exceptionally well 
established, developed and implemented safety culture and SMA 
policies, i.e. everyone knows when, how, and who is responsible for 
involving SR&QA, new people are trained in it, PMs held 
accountable for it - it is the way they do business. 
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S&MA considerations for R&T Projects 

(Right tools, right time, right extent) 

• Criteria for determining S&MA level of involvement 

[e.g. Pure ''paper studies" or data mining may not need S&MA involvement but 
awareness may still be needed ] 

- Current TRL level 

- Difficulty scale to advance technology ("is it a hard nut to crack?"), complexity 

- Risks & risk tolerance (analyzing/managing uncertainties) 

- $$ 

- Verifiable outcome 

. - Damage likelihood 

- Impact of failed research. (including public opinion) 

• • • 

- Who cares and how much do they care? 

- Potential for future growth/development 

- Ability to test . * 

- Path to build with reproducibility 

? 

- "Period of performance" drivers? 


CRTS Team Oil Buef 


13 



Defining Questions 


• Questions for Researchers 

1 . What are you trying to do? Articulate your objectives and explain the science and technology, please. 

2. Is there risk to the public, personnel, major NASA equipment, or Facilities? 

3. Could SMA support your efforts to assure that you can rely on the results of your investigations with: e.g. 
safety risk evaluation, calibration of measurement tools, contamination control, ESD control, configuration 
management, statistical design of experiments analyses 

4. If this technology eventually transitions to an operational technology for NASA, what Safety, Reliability, 
Maintainability and QA considerations may need to be addressed? 

• Questions for SMA 

1 . Do we understand what is trying to be accomplished for this experiment or development effort? 

2. Is there a current potential safety risk? 

3. Will S&MA involvement in the technology now be likely to reduce time, effort, and improve safety and 
“success”? 

4. What SRM and QA challenges may this technology or concept pose for SMA during development of the 
operational versions (TRL 8 and above) - so, what does S&MA need to do to be prepared to accept 
incorporation of this technology? 

5. Can SMA involvement help determine or improve the worthiness and acceptability of technology for 
infusion into Flight or Operational systems from a reliability, maintainability, safety and quality 
perspective? 

6. What is the potential for this technology to improve SRM and QA characteristics over the current State 
of the Art? 
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Challenges - Change the Paradigm 


• Inconsistent approach to R&TD at Centers and now with OCT directions and 
terminologies 

• Need to get visibility of OSMA requirements and processes in the project 
management documents, not just buried in our own documents 

• R&T sees us as potential “road blocks” and have the misconception we are 
there to stop projects - SMA is seen as a having a “cop mentality” 

- We are seen as one size fits all 

- We are seen as being very costly and slow 

• OCT wants to tell us when they think we should be involved, otherwise we 
should not be concerned with what they are doing 

• We need early involvement and awareness to avoid becoming a roadblock at 
infusion. 

- We need to understand the science, technologies, complexities, and risks in order to 
advise the R&T (including OCT supported Projects) communities of what SR&QAcan 
do for them. 

• “Bringing SR&QA in” does not mean we are going to impose anything!!!! 
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Summary 


• Many NASA processes already exist that cover much of what is needed for the 
lower and upper TRL levels, the mid level or “Game Changing" technologies 
though need improved approaches. 

• TRL levels are not enough to determine SMA’s roles and level of involvement 

• Improved communication, understanding and cooperation is needed between 
SMAand OCT, the Mission Directorates, and Center Researchers and 
Technologists and the project managers of R&TD projects. 

- Improved SMA focus on R&TD via selected, multi-tiered POCs 

- SMA needs better awareness of and participation in research selection, w. focus on 
transition to flight demonstrations and eventual incorporation into NASA systems 

• Other successful organizations have built a culture of safety and level of quality 
expectation in the facilities, projects, and personnel involved in R&T, for both 
internal and external projects. NASA can improve in this area as well. It does 
take time to instill this culture. 
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Summary (Cont.) 


• We need to “Change the Paradigm” from Flight Orientation: 

- Create Training and Awareness Campaigns 

- Make sure Researchers are aware of the benefits of all aspects of S&MA 
contributions 

- Increase visibility and presence without the “cop" paradigm 

- Make sure S&MA personnel are aware of the differences in our roles and 

. responsibilities in an R&T environment 

- Increase SMA visibility and presence without the “cop” paradigm 


17 








